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Abstract Body 


Background/Context: 

Several studies suggest that values-affirmation can serve as a simple, yet powerful, tool for 
dramatically reducing achievement gaps. Cohen et al. (2006; 2009) find a 40 percent reduction in 
the black- white achievement gap among 7 th grade students; Sherman et al. (2013) find a 20 to 30 
percent reduction in the Latino-white achievement gap among 7 th grade students; and Miyake et 
al. (2010) find a 60 percent reduction in the female-male achievement gap among college 
physics students. Yet, despite this demonstrated potential for values-affirmation to bring about 
powerful changes, salutary effects are not always obtained - even in randomized trials (Bonnan 
2012; Kost-Smith et al. 2012). 

Because subtle variations in implementation procedures may explain some of the 
variation in these findings, it is crucial for researchers to measure the fidelity with which 
interventions are implemented. Yet, this begs the question, fidelity to what? Fidelity analyses 
require explication of a ‘gold-standard’ intervention, a theoretical ideal that specifies the number 
and frequency of treatments, necessary participant responses and engagement, and important 
contextual conditions - in addition to defining a process of deployment against which the real- 
world intervention can be compared (Hulleman and Cordray 2009). Based on the intervention 
design of Cohen and colleagues (2006) and a broader assessment of the literature on values 
affirmation, we develop a rubric for assessing the fidelity of implementation and apply it to data 
from the MWAP research program. 

Our aim is part of a growing focus in social sciences as researchers have become 
increasingly aware that translating promising research into successful policy requires careful 
attention to and measurement of fidelity of implementation (e.g. Hulleman and Cordray 2009). 
The more complicated the ideal intervention, the greater the opportunity for the intervention to 
fail and the more important it is that researchers both develop facilitation strategies (training, 
monitoring, support, feedback, differentiation etc.) to improve fidelity and enact a plan to assess 
adherence to those protocols (Carroll et al. 2007). While the values affirmation does not seem to 
be a complicated instrument, the salience of contextual factors, the fact that teachers must serve 
as administrators, and the many ways in which the effect of the treatment can be ‘broken’ present 
both a compelling case for the development of facilitation strategies and the necessity of a 
detailed fidelity evaluation. 

Definition and evaluation of fidelity is key to successfully bringing interventions to scale 
in the field. Fidelity of implementation establishes internal validity and helps to distinguish 
between opportunities for innovation and adaptation and the critical components of an 
intervention, to which implementers must maintain fidelity to the original design (O’Donnell 
2008). Further, fidelity of implementation is likely to be of particular import with regard to social 
psychological classroom interventions since there are many, subtle ways in which the values 
affirmation can - in theory - be undermined or altered (Yeager and Walton 2011). 

Purpose / Objective / Research Question / Focus of Study: 

Two purposes drive this work. Firstly, fidelity has not been an explicit consideration in most 
prior values-affirmation work; and so an ideal classroom-administered values-affirmation has not 
been fully specified. We employ a critical components approach to identify the most crucial 
elements of a classroom-administered values-affirmation intervention. Through careful study of 
prior work, we identify intervention design (the intervention is tailored to fit the students’ 
capabilities and needs), teacher-centered delivery (teachers administer the intervention as 
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instructed), stealth (teachers and students view the intervention as nonnal and do not view it as 
an assessment or as research), timing (early in the year, and proximate to stressful events like 
exams), and student engagement (students follow instructions and affirm themselves) as the 
critical components of implementation fidelity that could serve to moderate the success of the 
intervention. It is our hope this framework will help measure fidelity in other studies. Secondly, 
our district-wide randomized trial of values-affirmation revealed important, but relatively 
modest, impacts compared to previous studies (Cohen et al. 2006, 2009; Miyake et al. 20010; 
Sherman et al, 2013). We utilize our metric to investigate fidelity of implementation, and its 
variability across schools and classrooms in our sample as one possible explanation for our 
comparatively modest effects. 

Setting: 

The MWAP intervention took place in an urban school district in the Midwest where National 
Assessment of Educational Progress (NAEP) scores show some of the largest achievement gaps 
between White and Black (Vanneman, Hamilton, Baldwin, Anderson and Rahman, 2009), and 
White and Hispanic students (Hemphill, Vanneman and Rahman, 2011). The intervention was 
implemented over two cohorts of 7th grade students across 1 1 schools with a total of 1,049 
students participating in Cohort 1 . 

Population / Participants / Subjects: 

All 7th grade students in the district were invited to participate in the trial, and 60% (N= 1 .049) 
consented to participate. The consent rate ranged from 55% to 75% across the 1 1 schools. There 
were 1048 total consented students by the administration of the second exercise in November. 
Consented students were randomized within schools to either the treatment or control condition 
for the duration of the study. Students who switched schools within the district throughout the 
year (2.2% of all consented) remained in their experimental group and are analyzed as members 
of the school in which they were initially observed. We also gathered qualitative data from 44 
teachers involved in the exercise implementation, across all 1 1 schools. The majority of 
educators included in the study were language arts teachers. 

Intervention / Program / Practice: 

The experiment was conducted during the 2011-2012 academic year. The self-affirmation 
intervention procedure follows Cohen and his colleagues (Cohen et al. 2006; Cohen et al. 2009; 
Cook et al. 2012; Sherman et al. 2013). Seventh grade students completed a short (15-20 minute) 
writing prompt as part of their language arts or homeroom class activities three to four times 
during the school year: once early in the school year (September or October), once prior to state 
examinations in November, once in the winter (January or February; four schools opted out of 
this administration), and once in the Spring (April or May). Consented students were randomized 
within school to either the treatment or control condition and non-consented students received a 
third neutral written exercise in a similar format. 

The first two treatment exercises were structured response questions in which students 
were asked to select two to three values that were important to them from a list. The third and 
fourth treatment exercises were open ended and prompted students to respond to a structured 
prompt about a specific value. In all exercise versions students were asked to reflect upon their 
choice of value or the value provided and free write a response for why that value is important to 
them. Non-consented and control condition students were assigned to one of two control 


SREE Spring 2015 Conference Abstract Template 


A-2 



versions. In one version of the control condition exercise students were given a similar list of 
values and asked to select those values that were not important to them. They were then 
prompted to reflect on why the values they selected may not be important to them and may be 
important to someone else. Students in the second control condition (which will be referred to as 
the Neutral condition) were given a prompt such as describing how they open their locker or get 
ready for school. Students in the same classroom would typically be given all three prompts. 
Students were provided with Spanish language and low-literacy versions of each exercise as 
needed to ensure full participation and engagement from all students. The fonnat of the exercises 
was changed slightly across implementations to minimize perceived repetition. 

Regular classroom teachers administered the exercise materials and were asked not to 
refer to the exercises as research. Teachers were blinded both to the experimental condition of 
individual students and of the specific experimental hypotheses about marginalized students and 
social identity threat. This was accomplished by creating cover sheets for student exercises that 
appeared identical. Participating teachers received training from MWAP staffers before the first 
exercise implementation to ensure proper delivery of the exercises. For each exercise 
implementation teachers were asked to complete a classroom debrief form. These reports asked 
teachers to record any unusual events, questions, or disruptions, as well as other classroom 
activities the day of the intervention. At the end of the school year, after all exercises had been 
completed, teachers were then asked to fill out a survey to gauge the stealth of intervention 
delivery. Teachers were asked to reflect on how the exercises were delivered and what language 
they had used in describing the exercises to students (i.e. as a test, as a free-write, etc.). 

Research Design: 

We first review the literature on values-affinnations in and out of classrooms to generate an ideal 
model that identifies the most critical components of the intervention and distinguishes those 
critical components from acceptable adaptations and those that must be avoided. From this 
review we identify five critical processes described above — intervention design (fit), low-stress, 
stealth, timing, and student engagement. We map both our own and Cohen et al.’s (2006; 2009) 
protocols onto this model and identify indicators for each of these five processes. These include 
some traditional fidelity metrics: timing, dosage, and participant responsiveness, all 
demonstrated moderators of values affirmation (Critcher et al. 2010; Shennan et al. 2013). We 
then use these indicators to generate student- and teacher-centered fidelity indices. 

Data Collection and Analysis: 

For each critical component — intervention design (or fit), low-stress, stealth, timing, and student 
engagement - we use qualitative and quantitative data to assess fidelity. Intervention design (fit) 
is assessed by monitoring the match between the interventions received by the student and the 
student’s particular linguistic needs and literacy levels. Characteristics of the students are 
measured by student and school-level administrative data, while use of adapted interventions 
(low-literacy, or Spanish-language) were noted by MWAP school liaisons. Data collected from 
classroom debrief reports, completed after each intervention, and end-of-year teacher surveys 
were used to assess adherence to the intended teacher-centered delivery protocols that may 
impact both the level of stress students feel and the “stealthiness” of the intervention delivery. 

We gathered descriptive data from the district about the timing of standardized tests to assess the 
adherence to the critical component of timing of the intervention. Finally, we conducted a 
content analysis of student responses to assess student participation and engagement in the 
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exercises. Student Fidelity to Instruction measures the extent to which student’s followed the 
explicit instructions of the exercise (i.e. identify a value and explain why it is important to self). 
Because the written instructions do not tell the student to “affirm values,” this measure acts as a 
reliability check on the exercises themselves. Fidelity to Treatment measures the extent to which 
student’s engaged in self-affirmation, per the treatment intention, and is coded as a binary 
variable as to whether or not the student’s exercise met the criteria for values-affirmation or 
attribute-affirmation. Teacher Fidelity to Delivery is generated by evaluating data gathered 
through classroom debrief fonns filled out by teacher after each exercise. 

Findings / Results: 

Overall, our implementations across the school year appear close to the ideal. However, in this 
scaled up intervention we find substantial variation in fidelity across schools and classrooms, 
particularly in terms of the teacher-centered delivery of the intervention (see Table 1). Using the 
index of Fidelity to Treatment (Self-Affirmation), we find high levels of fidelity to the cognitive 
process of values affirmation among treated students (see Figure 1). Further, we find that 
variation in student engagement exists regardless of condition assignment. We find notable 
declines in both student and teacher engagement with the exercise by the fourth affirmation 
exercise. We believe that these results could indicate a dampening effect on the impact of the 
intervention in our sample. We also identify a tension between the critical components of the 
intervention; fidelity to the critical component of stealth can ultimately compromise the 
maintenance of a low-stress environment for students, which may jeopardize the investment on 
the part of students (see Table 2). This internal conflict may explain the variation in teacher 
delivery that took place, despite the detailed training provided. 

Conclusions: 

Our measurements are limited by the realities of scaling up a research project into the everyday 
classrooms of teachers. However, even in the face of these challenges, we believe that our results 
make a strong case for future implementations to take fidelity into account. Assessment of 
fidelity of implementation using a critical components framework will ensure better 
understanding of variation in the impacts of values-affinnations. We also believe our data 
support integrating teachers more fully into the process of delivery in order to resolve the 
conflict between teacher blindness on one hand and stealth and normalcy for students on the 
other-as this internal contradiction was implicated in the variation in teacher delivery. We 
strongly caution that the integration of these written interventions into regular curriculum must 
be handled carefully. Lastly, even with the threats to the fidelity of delivery and stealth we find 
significant impacts of the intervention on the intended audience (Bonnan & Grigg, 2014). These 
significant positive impacts despite low fidelity of some critical components suggests that there 
may be aspects of delivery and stealth that are less important as they are defined and understood 
in the original intervention protocol. 
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Appendix B. Tables and Figures 

Not included in page count. 


Table 1: Exercise Fidelity Reports (Questions Relate to Stress and Stealth Components) 



Overall (%) 

High Student 
Affirm (%) 

Low Student 
Affirm (%) 

High Student 
FTI-T (%) 

Low Student 
FTI-T (%) 

Classroom Disruption 

62 

63 

58 

69 

59 

Noticed Difference in 
Exercises 

20 

24 

39 

20 

25 

Student Asked about 
Research 

18 

20 

11 

19 

18 

Average Teacher 
Compliance 

72 

83 

71 

64 

75 

N= 275 

275 reports 

7 schools 

4 schools 

8 schools 

3 schools 


Table 2: End of Year Teacher Survey Responses (Questions Relate to Stress and Stealth 
Components) 


Overall (%) 


27 

33 

58 


Described as Good for You 
Identified as Research 
Presented as a Normal 
Activity 

N= 26 
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Figure 1: Fidelity to Treatment 
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